The invention relates to the field of machine (e.g., computer) vision and its application to the optimization and utilization of human activity and/or interaction within an enterprise (e.g., a business, neighborhood, home, other region or area of concern) to monitor events and aid in the automated or semi-automated decision making process to manage the enterprise efficiently and responsibly.
Because of the proliferation of both computer and electronic sensor technology in the past 20 years, and owing to persistent fascination with having a machine (e.g., computers, but not limited to) provide functions that normally require human intelligence, the field of machine vision has matured significantly in both theory and practice in recent years. This maturation has been accompanied by many demonstrations and publications of demonstrations of using machine vision algorithms running on commercially available computers and using commercially available sensors (e.g., visible light focal plane arrays) to deduce sensor data events (e.g., motion or other change), to separate human related events from other events, to recognize objects that are both moving and stationary in sensor data and to recognize specific activities implied by the events deduced from the sensor data.
Most of this activity in applied and theoretical machine (e.g., computer) vision has been directed at the advanced areas of robotics and biometrics (the positive identification of a person), in which attempts are made to mimic the rich perception capabilities of humans for understanding their own interaction with their surroundings or in being able to deduce the unique personal features (e.g., facial characteristics, movement, gait, etc.) of other people to aid in uniquely identifying them. These are, of course, significantly difficult problems to address and remain an ongoing challenge for researchers in machine vision.
However, as a result of much of the research community having focused their machine vision efforts on the long term objectives of human perception and biometric recognition, and since these, with only a few exceptions, remain largely unsolved at a level suitable for full commercial exploitation, the significant advances made to date have not found application in commercial products, even though there are applications for which the machine vision technology (as available in the open literature and open source code processing components) can currently be utilized to provide significant commercial value to retail, wholesale, manufacturing concerns and municipalities as well.
What is missing to date is a system design and architecture that integrates elements of machine vision algorithms and commercially relevant human activities with an enterprise (e.g., a business, neighborhood, home, other region or area of concern) network and communications infrastructure, so that the progress in machine vision to date for observing objects, basic human activities and fundamental events (e.g., motions or other changes) can be used to provide valuable and timely information (e.g., situational awareness) about the ongoing operation of the enterprise.
The invention disclosed, herein, articulates a novel system design that uses sensor data (e.g., machine vision) events and combines them with key activity identifiers (e.g., specific human activity, animal activity, or interactions between humans, animals, machines, etc.) so that event data from sensors can be made available to decision logic for both real time (e.g., instant message, as an example, but not limited to) and delayed (e.g., database server, data mining application service, but not limited to) notification and/or recording of events and interrelationships of interest to a commercial, municipal type public enterprise or private enterprise, many of which enterprises also have a customer service oriented environment. Furthermore, the system architecture of this invention enables the application of machine vision technology to automated or semi-automated enterprise situational awareness for deducing and optimizing the value (e.g., commercial value, but not limited to) of human interaction with other human subjects and/or animals (as these are commonly found in many enterprises and are known to be associated with human activity in some settings) and/or enterprise equipment in use by or associated with the human subjects.
What this invention does that has not been done to date is to provide a automated or semi-automated system design that gives non-practitioners in the field of machine vision and statistical data analysis access to advanced machine vision and statistical data analysis capabilities for use in improving or enhancing the effectiveness of an enterprise and its management in which human activity figures prominently in either the use of the enterprise itself (e.g., a residence or office building) or the use of the enterprise for production, sale or purchase of saleable product (e.g., an electronics assembly facility, a restaurant, a retail consumer goods store, etc. to name a few, but not limited to).
As used in this disclosure, an event is a phenomenon that occurs in the scene of particular interest viewed by the sensor that may be of particular interest to a particular user of the invention. Since the purpose of the invention is to provide situational awareness to a user having particular enterprise-related concerns, not all events result in a communication (e.g. message) to the user. Rather, events are communicated to a user only insomuch as they are similar to events that the user determines are important, or that are deemed important by the invention through evidence of significance implied by the temporal and/or spatial behavior of event data (e.g., an event not easily confused with mere randomness in sensor data occurs at a time that is highly unusual, an event occurs repeatedly over time, or repeatedly in the same place, etc., to name a few examples).
For this disclosure, these events are key activities. From this it follows that a key activity message that is useful in enterprise management will contain any or all of five elements: 1) what object is moving or changing (person, animal, vehicle, machine, other inanimate object) 2) what behavior the moving or changing object is engaged in, 3) where the object is located, 4) when the motion or change is occurring and 5) measurable interpretation of the change in phenomena or activity in order to make a decision by a machine or a human for a purpose.
Events that may be of interest to a retail store user would, for example, include the length of a line at the cash register exceeding three people (as an example), or some other number that the user deemed unacceptably high. An example of an event of interest to a neighborhood organization, for instance, would be a vehicle that appears in an alley several nights consecutively when people in the neighborhood are sleeping. An example event that might be of interest for roadways and transportation concerns is the number of people in line waiting to cross a busy street at which a traffic signal has been placed.